System and method for updating vehicle operation based on remote intervention

ABSTRACT

Technologies disclosed relate to a remote intervention system for the operation of a vehicle, which can be an autonomous vehicle, a vehicle that includes driver assist features, a vehicle used for ride sharing services or the like. The system includes a vehicle sending a request for remote intervention to a remote operator when the operation of the vehicle is suspended. The request for remote intervention can include a request for object identification or a request for decision confirmation. The vehicle can update vehicle operation based in part on vehicle-based sensor data and a response to the remote intervention request from the remote operator. The remote operator can be a human operator or an AI operator.

INCORPORATION BY REFERENCE TO ANY PRIORITY APPLICATIONS

This application claims the benefit of U.S. Patent Application No.62/786,261, filed Dec. 28, 2018, which is hereby incorporated byreference in their entirety.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentand/or the patent disclosure as it appears in the United States Patentand Trademark Office patent file and/or records, but otherwise reservesall copyrights whatsoever.

BACKGROUND

Vehicles—such as vehicles used for ride-sharing purposes, vehicles thatprovide driver-assist functionality, and/or automated or autonomousvehicles (AVs)—may obtain and process sensor data using an on-board dataprocessing system to perform a variety of functions. For example,functions can include determining and/or displaying navigational routes,identifying road signs, detecting objects and/or road obstructions,controlling vehicle operation, and/or the like. However, the operationof an autonomous vehicle or a vehicle with driver-assist functionalitymay become suspended or disabled if the on-board data processing systemcannot process the sensor data or cannot determine the proper course ofaction to take due to unexpected road conditions.

SUMMARY

The present disclosure provides systems and methods of providing remoteintervention to vehicles, such as autonomous vehicle, a vehicle thatincludes driver-assist functionality, or a vehicle used for ride sharingservices, when the operation of the vehicle is suspended or disabled.The vehicle can transmit visual data detected by one or more sensors onthe vehicle to a remote operator. The remote operator can be a humanoperator or an artificial intelligence (AI) operator. In someembodiments, the remote operator can provide direct command ordecision-making assistance to the vehicle upon analyzing the visual datatransmitted by the vehicle. In some embodiments, the vehicle can retaincontrol of the vehicle and output a control signal to update vehicleoperation based on aggregated information including a decision-makingassistance response from the remote operator and inputs from the one ormore sensors.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates a block diagram of a networked vehicle environmentin which one or more vehicles and/or one or more user devices interactwith a server via a network, according to one embodiment.

FIG. 1B illustrates a block diagram showing the vehicle of FIG. 1A incommunication with one or more other vehicles and/or the server of FIG.1A, according to one embodiment.

FIGS. 2A-2C illustrate schematically examples when a vehicle may bedisabled or its operation may be suspended.

FIG. 3 illustrates schematically an example system for a remote operatorto provide remote intervention to a vehicle.

FIG. 4 illustrates a flow chart for an example process of requestingremote intervention by a vehicle.

FIG. 5 illustrates a flow chart for an example process of providing aremote intervention response to a vehicle by a remote server.

FIG. 6A illustrates a flow chart of an example process of a vehiclereceiving remote intervention.

FIG. 6B illustrates a flow chart of an example process of a remoteoperator providing remote intervention to a vehicle.

FIGS. 7A-7B illustrate flow charts of examples of remote interventionprovided by a remote server.

FIG. 8A illustrates an example decision-making block diagram of anonboard data processing system of a vehicle.

FIG. 8B illustrates a flow chart of an example process by a vehiclemaking decisions about movements of the vehicle based on a remoteintervention response from a remote server and vehicle-based sensordata.

FIG. 9 illustrates a flow of an example process performed by anartificial intelligence (AI) operator in providing remote interventionto a vehicle.

DETAILED DESCRIPTION

Detailed descriptions and examples of systems and methods according toone or more illustrative embodiments of the present disclosure may befound in the section entitled Example Remote Intervention Systems(including its subsections), as well as in the section entitled ExampleEmbodiments, and also in FIGS. 2A-9 herein. Furthermore, components andfunctionality for providing remote intervention to a vehicle may beconfigured and/or incorporated into the networked vehicle environment100 described herein in FIGS. 1A-1B.

Various embodiments described herein are intimately tied to, enabled by,and would not exist except for, vehicle and/or computer technology. Forexample, providing remote intervention to a vehicle described herein inreference to various embodiments cannot reasonably be performed byhumans alone, without the vehicle and/or computer technology upon whichthey are implemented.

Networked Vehicle Environment

FIG. 1A illustrates a block diagram of a networked vehicle environment100 in which one or more vehicles 120 and/or one or more user devices102 interact with a server 130 via a network 110, according to oneembodiment. For example, the vehicles 120 may be equipped to provideride-sharing and/or other location-based services, to assist drivers incontrolling vehicle operation (e.g., via various driver-assist features,such as adaptive and/or regular cruise control, adaptive headlightcontrol, anti-lock braking, automatic parking, night vision, blind spotmonitor, collision avoidance, crosswind stabilization, driver drowsinessdetection, driver monitoring system, emergency driver assistant,intersection assistant, hill descent control, intelligent speedadaptation, lane centering, lane departure warning, forward, rear,and/or side parking sensors, pedestrian detection, rain sensor, surroundview system, tire pressure monitor, traffic sign recognition, turningassistant, wrong-way driving warning, traffic condition alerts, etc.),and/or to fully control vehicle operation. Thus, the vehicles 120 can beregular gasoline, natural gas, biofuel, electric, hydrogen, etc.vehicles configured to offer ride-sharing and/or other location-basedservices, vehicles that provide driver-assist functionality (e.g., oneor more of the driver-assist features described herein), and/orautomated or autonomous vehicles (AVs). The vehicles 120 can beautomobiles, trucks, vans, buses, motorcycles, scooters, bicycles,and/or any other motorized vehicle.

The server 130 can communicate with the vehicles 120 to obtain vehicledata, such as route data, sensor data, perception data, vehicle 120control data, vehicle 120 component fault and/or failure data, etc. Theserver 130 can process and store the vehicle data for use in otheroperations performed by the server 130 and/or another computing system(not shown). Such operations can include running diagnostic models toidentify vehicle 120 operational issues (e.g., the cause of vehicle 120navigational errors, unusual sensor readings, an object not beingidentified, vehicle 120 component failure, etc.); running models tosimulate vehicle 120 performance given a set of variables; identifyingobjects that cannot be identified by a vehicle 120, generating controlinstructions that, when executed by a vehicle 120, cause the vehicle 120to drive and/or maneuver in a certain manner along a specified path;and/or the like.

The server 130 can also transmit data to the vehicles 120. For example,the server 130 can transmit map data, firmware and/or software updates,vehicle 120 control instructions, an identification of an object thatcould not otherwise be identified by a vehicle 120, passenger pickupinformation, traffic data, and/or the like.

In addition to communicating with one or more vehicles 120, the server130 can communicate with one or more user devices 102. In particular,the server 130 can provide a network service to enable a user torequest, via an application running on a user device 102, location-basedservices (e.g., transportation services, such as ride-sharing services).For example, the user devices 102 can correspond to a computing device,such as a smart phone, tablet, laptop, smart watch, or any other devicethat can communicate over the network 110 with the server 130. In theembodiment, a user device 102 executes an application, such as a mobileapplication, that the user operating the user device 102 can use tointeract with the server 130. For example, the user device 102 cancommunicate with the server 130 to provide location data and/or queriesto the server 130, to receive map-related data and/or directions fromthe server 130, and/or the like.

The server 130 can process requests and/or other data received from userdevices 102 to identify service providers (e.g., vehicle 120 drivers) toprovide the requested services for the users. In addition, the server130 can receive data—such as user trip pickup or destination data, userlocation query data, etc.—based on which the server 130 identifies aregion, an address, and/or other location associated with the varioususers. The server 130 can then use the identified location to provideservices providers and/or users with directions to a determined pickuplocation.

The application running on the user device 102 may be created and/ormade available by the same entity responsible for the server 130.Alternatively, the application running on the user device 102 can be athird-party application that includes features (e.g., an applicationprogramming interface or software development kit) that enablescommunications with the server 130.

A single server 130 is illustrated in FIG. 1A for simplicity and ease ofexplanation. It is appreciated, however, that the server 130 may be asingle computing device, or may include multiple distinct computingdevices logically or physically grouped together to collectively operateas a server system. The components of the server 130 can be implementedin application-specific hardware (e.g., a server computing device withone or more ASICs) such that no software is necessary, or as acombination of hardware and software. In addition, the modules andcomponents of the server 130 can be combined on one server computingdevice or separated individually or into groups on several servercomputing devices. In some embodiments, the server 130 may includeadditional or fewer components than illustrated in FIG. 1A.

The network 110 includes any wired network, wireless network, orcombination thereof. For example, the network 110 may be a personal areanetwork, local area network, wide area network, over-the-air broadcastnetwork (e.g., for radio or television), cable network, satellitenetwork, cellular telephone network, or combination thereof. As afurther example, the network 110 may be a publicly accessible network oflinked networks, possibly operated by various distinct parties, such asthe Internet. In some embodiments, the network 110 may be a private orsemi-private network, such as a corporate or university intranet. Thenetwork 110 may include one or more wireless networks, such as a GlobalSystem for Mobile Communications (GSM) network, a Code Division MultipleAccess (CDMA) network, a Long Term Evolution (LTE) network, or any othertype of wireless network. The network 110 can use protocols andcomponents for communicating via the Internet or any of the otheraforementioned types of networks. For example, the protocols used by thenetwork 110 may include Hypertext Transfer Protocol (HTTP), HTTP Secure(HTTPS), Message Queue Telemetry Transport (MQTT), ConstrainedApplication Protocol (CoAP), and the like. Protocols and components forcommunicating via the Internet or any of the other aforementioned typesof communication networks are well known to those skilled in the artand, thus, are not described in more detail herein.

The server 130 can include a navigation unit 140, a vehicle dataprocessing unit 145, and a data store 150. The navigation unit 140 canassist with location-based services. For example, the navigation unit140 can facilitate the transportation of a user (also referred to hereinas a “rider”) and/or an object (e.g., food, packages, etc.) by anotheruser (also referred to herein as a “driver”) from a first location (alsoreferred to herein as a “pickup location”) to a second location (alsoreferred to herein as a “destination location”). The navigation unit 140may facilitate user and/or object transportation by providing map and/ornavigation instructions to an application running on a user device 102of a rider, to an application running on a user device 102 of a driver,and/or to a navigational system running on a vehicle 120.

As an example, the navigation unit 140 can include a matching service(not shown) that pairs a rider requesting a trip from a pickup locationto a destination location with a driver that can complete the trip. Thematching service may interact with an application running on the userdevice 102 of the rider and/or an application running on the user device102 of the driver to establish the trip for the rider and/or to processpayment from the rider to the driver.

The navigation unit 140 can also communicate with the applicationrunning on the user device 102 of the driver during the trip to obtaintrip location information from the user device 102 (e.g., via a globalposition system (GPS) component coupled to and/or embedded within theuser device 102) and provide navigation directions to the applicationthat aid the driver in traveling from the current location of the driverto the destination location. The navigation unit 140 can also direct thedriver to various geographic locations or points of interest, regardlessof whether the driver is carrying a rider.

The vehicle data processing unit 145 can be configured to supportvehicle 120 driver-assist features and/or to support autonomous driving.For example, the vehicle data processing unit 145 can generate and/ortransmit to a vehicle 120 map data, run diagnostic models to identifyvehicle 120 operational issues, run models to simulate vehicle 120performance given a set of variables, use vehicle data provided by avehicle 120 to identify an object and transmit an identification of theobject to the vehicle 120, generate and/or transmit to a vehicle 120vehicle 120 control instructions, and/or the like.

The data store 150 can store various types of data used by thenavigation unit 140, the vehicle data processing unit 145, the userdevices 102, and/or the vehicles 120. For example, the data store 150can store user data 152, map data 154, search data 156, and log data158.

The user data 152 may include information on some or all of the usersregistered with a location-based service, such as drivers and riders.The information may include, for example, usernames, passwords, names,addresses, billing information, data associated with prior trips takenor serviced by a user, user rating information, user loyalty programinformation, and/or the like.

The map data 154 may include high definition (HD) maps generated fromsensors (e.g., light detection and ranging (LiDAR) sensors, radiodetection and ranging (RADAR) sensors, infrared cameras, visible lightcameras, stereo cameras, an inertial measurement unit (IMU), etc.),satellite imagery, optical character recognition (OCR) performed oncaptured street images (e.g., to identify names of streets, to identifystreet sign text, to identify names of points of interest, etc.), etc.;information used to calculate routes; information used to render 2Dand/or 3D graphical maps; and/or the like. For example, the map data 154can include elements like the layout of streets and intersections,bridges (e.g., including information on the height and/or width ofbridges over streets), off-ramps, buildings, parking structure entrancesand exits (e.g., including information on the height and/or width of thevehicle entrances and/or exits), the placement of street signs and stoplights, emergency turnoffs, points of interest (e.g., parks,restaurants, fuel stations, attractions, landmarks, etc., and associatednames), road markings (e.g., centerline markings dividing lanes ofopposing traffic, lane markings, stop lines, left turn guide lines,right turn guide lines, crosswalks, bus lane markings, bike lanemarkings, island marking, pavement text, highway exist and entrancemarkings, etc.), curbs, rail lines, waterways, turning radiuses and/orangles of left and right turns, the distance and dimensions of roadfeatures, the placement of barriers between two-way traffic, and/or thelike, along with the elements' associated geographical locations (e.g.,geographical coordinates). The map data 154 can also include referencedata, such as real-time and/or historical traffic information, currentand/or predicted weather conditions, road work information, informationregarding laws and regulations (e.g., speed limits, whether right turnson red lights are permitted or prohibited, whether U-turns are permittedor prohibited, permitted direction of travel, and/or the like), newsevents, and/or the like.

While the map data 154 is illustrated as being stored in the data store150 of the server 130, this is not meant to be limiting. For example,the server 130 can transmit the map data 154 to a vehicle 120 forstorage therein (e.g., in the data store 129, described below).

The search data 156 can include searches entered by various users in thepast. For example, the search data 156 can include textual searches forpickup and/or destination locations. The searches can be for specificaddresses, geographical locations, names associated with a geographicallocation (e.g., name of a park, restaurant, fuel station, attraction,landmark, etc.), etc.

The log data 158 can include vehicle data provided by one or morevehicles 120. For example, the vehicle data can include route data,sensor data, perception data, vehicle 120 control data, vehicle 120component fault and/or failure data, etc.

FIG. 1B illustrates a block diagram showing the vehicle 120 of FIG. 1Ain communication with one or more other vehicles 170A-N and/or theserver 130 of FIG. 1A, according to one embodiment. As illustrated inFIG. 1B, the vehicle 120 can include various components and/or datastores. For example, the vehicle 120 can include a sensor array 121, acommunications array 122, a data processing system 123, a communicationsystem 124, an interior interface system 125, a vehicle control system126, operative systems 127, a mapping engine 128, and/or a data store129.

Communications 180 may be transmitted and/or received between thevehicle 120, one or more vehicles 170A-N, and/or the server 130. Theserver 130 can transmit and/or receive data from the vehicle 120 asdescribed above with respect to FIG. 1A. For example, the server 130 cantransmit vehicle control instructions or commands (e.g., ascommunications 180) to the vehicle 120. The vehicle control instructionscan be received by the communications array 122 (e.g., an array of oneor more antennas configured to transmit and/or receive wirelesssignals), which is operated by the communication system 124 (e.g., atransceiver). The communication system 124 can transmit the vehiclecontrol instructions to the vehicle control system 126, which canoperate the acceleration, steering, braking, lights, signals, and otheroperative systems 127 of the vehicle 120 in order to drive and/ormaneuver the vehicle 120 and/or assist a driver in driving and/ormaneuvering the vehicle 120 through road traffic to destinationlocations specified by the vehicle control instructions.

As an example, the vehicle control instructions can include route data163, which can be processed by the vehicle control system 126 tomaneuver the vehicle 120 and/or assist a driver in maneuvering thevehicle 120 along a given route (e.g., an optimized route calculated bythe server 130 and/or the mapping engine 128) to the specifieddestination location. In processing the route data 163, the vehiclecontrol system 126 can generate control commands 164 for execution bythe operative systems 127 (e.g., acceleration, steering, braking,maneuvering, reversing, etc.) to cause the vehicle 120 to travel alongthe route to the destination location and/or to assist a driver inmaneuvering the vehicle 120 along the route to the destination location.

A destination location 166 may be specified by the server 130 based onuser requests (e.g., pickup requests, delivery requests, etc.)transmitted from applications running on user devices 102. Alternativelyor in addition, a passenger and/or driver of the vehicle 120 can provideuser input(s) 169 through an interior interface system 125 (e.g., avehicle navigation system) to provide a destination location 166. Insome embodiments, the vehicle control system 126 can transmit theinputted destination location 166 and/or a current location of thevehicle 120 (e.g., as a GPS data packet) as a communication 180 to theserver 130 via the communication system 124 and the communications array122. The server 130 (e.g., the navigation unit 140) can use the currentlocation of the vehicle 120 and/or the inputted destination location 166to perform an optimization operation to determine an optimal route forthe vehicle 120 to travel to the destination location 166. Route data163 that includes the optimal route can be transmitted from the server130 to the vehicle control system 126 via the communications array 122and the communication system 124. As a result of receiving the routedata 163, the vehicle control system 126 can cause the operative systems127 to maneuver the vehicle 120 through traffic to the destinationlocation 166 along the optimal route, assist a driver in maneuvering thevehicle 120 through traffic to the destination location 166 along theoptimal route, and/or cause the interior interface system 125 to displayand/or present instructions for maneuvering the vehicle 120 throughtraffic to the destination location 166 along the optimal route.

Alternatively or in addition, the route data 163 includes the optimalroute and the vehicle control system 126 automatically inputs the routedata 163 into the mapping engine 128. The mapping engine 128 cangenerate map data 165 using the optimal route (e.g., generate a mapshowing the optimal route and/or instructions for taking the optimalroute) and provide the map data 165 to the interior interface system 125(e.g., via the vehicle control system 126) for display. The map data 165may include information derived from the map data 154 stored in the datastore 150 on the server 130. The displayed map data 165 can indicate anestimated time of arrival and/or show the progress of the vehicle 120along the optimal route. The displayed map data 165 can also includeindicators, such as reroute commands, emergency notifications, road workinformation, real-time traffic data, current weather conditions,information regarding laws and regulations (e.g., speed limits, whetherright turns on red lights are permitted or prohibited, where U-turns arepermitted or prohibited, permitted direction of travel, etc.), newsevents, and/or the like.

The user input 169 can also be a request to access a network (e.g., thenetwork 110). In response to such a request, the interior interfacesystem 125 can generate an access request 168, which can be processed bythe communication system 124 to configure the communications array 122to transmit and/or receive data corresponding to a user's interactionwith the interior interface system 125 and/or with a user device 102 incommunication with the interior interface system 125 (e.g., a userdevice 102 connected to the interior interface system 125 via a wirelessconnection). For example, the vehicle 120 can include on-board Wi-Fi,which the passenger(s) and/or driver can access to send and/or receiveemails and/or text messages, stream audio and/or video content, browsecontent pages (e.g., network pages, web pages, etc.), and/or accessapplications that use network access. Based on user interactions, theinterior interface system 125 can receive content 167 via the network110, the communications array 122, and/or the communication system 124.The communication system 124 can dynamically manage network access toavoid or minimize disruption of the transmission of the content 167.

The sensor array 121 can include any number of one or more types ofsensors, such as a satellite-radio navigation system (e.g., GPS), aLiDAR sensor, a landscape sensor (e.g., a radar sensor), an IMU, acamera (e.g., an infrared camera, a visible light camera, stereocameras, etc.), a Wi-Fi detection system, a cellular communicationsystem, an inter-vehicle communication system, a road sensorcommunication system, feature sensors, proximity sensors (e.g.,infrared, electromagnetic, photoelectric, etc.), distance sensors, depthsensors, and/or the like. The satellite-radio navigation system maycompute the current position (e.g., within a range of 1-10 meters) ofthe vehicle 120 based on an analysis of signals received from aconstellation of satellites.

The LiDAR sensor, the radar sensor, and/or any other similar types ofsensors can be used to detect the vehicle 120 surroundings while thevehicle 120 is in motion or about to begin motion. For example, theLiDAR sensor may be used to bounce multiple laser beams off approachingobjects to assess their distance and to provide accurate 3D informationon the surrounding environment. The data obtained from the LiDAR sensormay be used in performing object identification, motion vectordetermination, collision prediction, and/or in implementing accidentavoidance processes. Optionally, the LiDAR sensor may provide a 360°view using a rotating, scanning mirror assembly. The LiDAR sensor mayoptionally be mounted on a roof of the vehicle 120.

The IMU may include X, Y, Z oriented gyroscopes and/or accelerometers.The IMU provides data on the rotational and linear motion of the vehicle120, which may be used to calculate the motion and position of thevehicle 120.

Cameras may be used to capture visual images of the environmentsurrounding the vehicle 120. Depending on the configuration and numberof cameras, the cameras may provide a 360° view around the vehicle 120.The images from the cameras may be used to read road markings (e.g.,lane markings), read street signs, detect objects, and/or the like.

The Wi-Fi detection system and/or the cellular communication system maybe used to perform triangulation with respect to Wi-Fi hot spots or celltowers respectively, to determine the position of the vehicle 120(optionally in conjunction with then satellite-radio navigation system).

The inter-vehicle communication system (which may include the Wi-Fidetection system, the cellular communication system, and/or thecommunications array 122) may be used to receive and/or transmit data tothe other vehicles 170A-N, such as current speed and/or locationcoordinates of the vehicle 120, time and/or location coordinatescorresponding to when deceleration is planned and the planned rate ofdeceleration, time and/or location coordinates when a stop operation isplanned, time and/or location coordinates when a lane change is plannedand direction of lane change, time and/or location coordinates when aturn operation is planned, time and/or location coordinates when aparking operation is planned, and/or the like.

The road sensor communication system (which may include the Wi-Fidetection system and/or the cellular communication system) may be usedto read information from road sensors (e.g., indicating the trafficspeed and/or traffic congestion) and/or traffic control devices (e.g.,traffic signals).

When a user requests transportation (e.g., via the application runningon the user device 102), the user may specify a specific destinationlocation. The origination location may be the current location of thevehicle 120, which may be determined using the satellite-radionavigation system installed in the vehicle (e.g., GPS, Galileo,BeiDou/COMPASS, DORIS, GLONASS, and/or other satellite-radio navigationsystem), a Wi-Fi positioning System, cell tower triangulation, and/orthe like. Optionally, the origination location may be specified by theuser via a user interface provided by the vehicle 120 (e.g., theinterior interface system 125) or via the user device 102 running theapplication. Optionally, the origination location may be automaticallydetermined from location information obtained from the user device 102.In addition to the origination location and destination location, one ormore waypoints may be specified, enabling multiple destinationlocations.

Raw sensor data 161 from the sensor array 121 can be processed by theon-board data processing system 123. The processed data 162 can then besent by the data processing system 123 to the vehicle control system126, and optionally sent to the server 130 via the communication system124 and the communications array 122.

The data store 129 can store map data (e.g., the map data 154) and/or asubset of the map data 154 (e.g., a portion of the map data 154corresponding to a general region in which the vehicle 120 is currentlylocated). In some embodiments, the vehicle 120 can use the sensor array121 to record updated map data along traveled routes, and transmit theupdated map data to the server 130 via the communication system 124 andthe communications array 122. The server 130 can then transmit theupdated map data to one or more of the vehicles 170A-N and/or furtherprocess the updated map data.

The data processing system 123 can provide continuous or near continuousprocessed data 162 to the vehicle control system 126 to respond topoint-to-point activity in the surroundings of the vehicle 120. Theprocessed data 162 can comprise comparisons between the raw sensor data161—which represents an operational environment of the vehicle 120, andwhich is continuously collected by the sensor array 121—and the map datastored in the data store 129. In an example, the data processing system123 is programmed with machine learning or other artificial intelligencecapabilities to enable the vehicle 120 to identify and respond toconditions, events, and/or potential hazards. In variations, the dataprocessing system 123 can continuously or nearly continuously compareraw sensor data 161 to stored map data in order to perform alocalization to continuously or nearly continuously determine a locationand/or orientation of the vehicle 120. Localization of the vehicle 120may allow the vehicle 120 to become aware of an instant location and/ororientation of the vehicle 120 in comparison to the stored map data inorder to maneuver the vehicle 120 on surface streets through trafficand/or assist a driver in maneuvering the vehicle 120 on surface streetsthrough traffic and identify and respond to potential hazards (e.g.,pedestrians) or local conditions, such as weather or traffic conditions.

Furthermore, localization can enable the vehicle 120 to tune or beamsteer the communications array 122 to maximize a communication linkquality and/or to minimize interference with other communications fromother vehicles 170A-N. For example, the communication system 124 canbeam steer a radiation patterns of the communications array 122 inresponse to network configuration commands received from the server 130.The data store 129 may store current network resource map data thatidentifies network base stations and/or other network sources thatprovide network connectivity. The network resource map data may indicatelocations of base stations and/or available network types (e.g., 3G, 4G,LTE, Wi-Fi, etc.) within a region in which the vehicle 120 is located.

While FIG. 1B describes certain operations as being performed by thevehicle 120 or the server 130, this is not meant to be limiting. Theoperations performed by the vehicle 120 and the server 130 as describedherein can be performed by either entity. For example, certainoperations normally performed by the server 130 (e.g., transmittingupdating map data to the vehicles 170A-N) may be performed by thevehicle 120 for load balancing purposes (e.g., to reduce the processingload of the server 130, to take advantage of spare processing capacityon the vehicle 120, etc.).

Furthermore, any of the vehicles 170A-N may include some or all of thecomponents of the vehicle 120 described herein. For example, a vehicle170A-N can include a communications array 122 to communicate with thevehicle 120 and/or the server 130.

Example Remote Intervention Systems

As described above, the vehicle 120, which can be an autonomous vehicle,a vehicle that includes driver-assist features, or a vehicle used forride-sharing services, can include a plurality of sensors (such asLiDAR, ultrasonic sensor, camera, and the like that are described above)to gather information (such as visual, audio, and the like) that canhelp its onboard data processing system 123 to make decisions about theoperation of the vehicle. However, in some circumstances, operation ofthe vehicle may remain suspended or disabled. FIGS. 2A-2B illustratecertain conditions in which the operation of a vehicle 200 (which can bethe vehicle 120 described above) can be suspended, such as due to thevehicle 200 encountering an indeterminate object when moving on aplanned route.

For example, as shown in FIG. 2A, an indeterminate object can include anunknown and/or unexpected obstacle 202 occupying a current lane 204 inwhich the vehicle 200 is traveling. The obstacle 202 can be detected bya sensor on the vehicle 200, such as the LiDAR 210, camera, Radar, anultrasonic sensor, or any other sensor disclosed herein. Examples ofsuch unknown and/or unexpected obstacles can include a parade, flooding,objects dropped in the current lane 204 (such as a piece of furniturefalling off from the back of a truck). A human driver, when encounteringsuch obstacles, may navigate a vehicle around those obstacles andcontinue toward a destination location. In contrast, the onboard dataprocessing system of the vehicle 200 may lack an understanding of thoseobstacles the way a human driver understands the obstacles. For safetyreasons, the vehicle 200 may treat those obstacles as indications forstopping or slowing down to a stop, sometimes for a prolonged period oftime until the obstacles are removed.

The vehicles may also rely on detecting marker objects (which can be anyof the objects disclosed herein, such as traffic lights, road signs,street markers, and the like) along a planned route to ascertain whetherthe vehicle is following the planned route. Certain marker objects maybe located outside the field of detection of the plurality of sensors onthe vehicle. As shown in FIG. 2B, the vehicle 200 traveling in the lane204 is expected to pass a marker object, such as a traffic light 206, onits planned route. The traffic light 206 can be located slightly off ofthe lane 204. A human driver may be able to determine the presence ofthe traffic light 206 by looking slightly to the right. However, as thetraffic light 206 is located outside the field of detection of thesensor(s) on the vehicle 200, such as the LiDAR 210, camera, Radar, anultrasonic sensor, or otherwise, the processing system of the vehiclewould interpret the sensor data as having a missing marker object. Themissing marker object can cause the vehicle 200 to be unable todetermine whether the vehicle 200 is traveling on its planned route. Forsafety reasons, the vehicle 200 may treat the missing marker object asan indication for stopping or slowing down to a stop, sometimes for aprolonged period of time, such as by a server connected to the vehicle(such as the server 130) confirming the presence of the marker object orconfirming that the vehicle is traveling on the planned route.

In some instances, the onboard data processing system of a vehicle mayneed additional authorization for certain maneuvers, such as switchinglanes. As shown in FIG. 2C, the vehicle 200 encounters a roadblock 206in the lane 204 traveled by the vehicle 200. The roadblock 206 canremain in the lane 204 for a prolonged period of time, for example, formore than one hour, two hours, half a day, or longer. A lane 208adjacent to the lane 204 is available. However, the lanes 204, 208 canbe separated by a solid line (for example, when the vehicle 200 hasentered a region before the cross-road where lane change is prohibitedby traffic rules). A human driver traveling on the lane 204 cancircumvent the roadblock 206 by switching to the adjacent lane 208.However, the processing system of the vehicle 200 may be programed tonever cross a solid line under any circumstances and may need additionalauthorization to cross over to the adjacent lane 208.

In situations such as illustrated in FIGS. 2A-2C, the operation of thevehicle 200 may remain suspended. The vehicle 200 may be stuck at itscurrent location for a prolonged period of time, such as one hour, twohours, half a day, or even longer. A service team may need to bedispatched to the vehicle 200 to retrieve the vehicle. Dispatching aservice team can be labor-intensive and expensive. Also, as describedabove, in the situation such as illustrated in FIGS. 2A-2C, the road isnot truly undrivable. Rather, the processing system of the vehicle 200may lack the ability to adapt to the road conditions such as illustratedin FIGS. 2A-2C the way a human driver could.

The present disclosure provides systems and methods for reducing theoccurrence of situations in which the operation of a vehicle issuspended when a human driver could have continued driving. FIG. 3illustrates schematically a vehicle 300 (which can be any of the vehicleexamples disclosed herein) receiving remote intervention from a remoteserver 350 (which can be the server 130 described above, or a differentserver). A remote operator 352 can be located at the remote server 350.The remote operator 352 can be a human operator or an AI operator. Whenthe operation of the vehicle 300 is suspended or disabled, such as insituations as illustrated in FIGS. 2A-2C, the vehicle 300 can send arequest for intervention to the remote server 350 via a wirelesstransmitter (using any of the wireless communication protocols describedabove) on the vehicle 300. The remote server 350 can provide a responseto the vehicle 300 after the remote operator 352 has reviewed thesituation in which the operation of the vehicle 300 is suspended. Theoperation of the vehicle 300 can be updated based at least in part onthe response.

FIG. 4 illustrates an example process 400 of requesting and receivingremote intervention by an onboard data processing system of a vehicledisclosed herein. At step 402, the onboard data processing system of thevehicle can receive and process signal inputs from the one or moresensors on the vehicle. At decision step 404, the onboard dataprocessing system can determine whether an object can be detected, forexample, a predetermined distance ahead of the vehicle and/or in thelane traveled by the vehicle. If no object can be detected, the onboarddata processing system can continue to monitor the surroundings byreturning to step 402. If an object can be detected, at decision step406, the onboard data processing system can determine whether a markerobject (such as a traffic light or a road sign) is expected to bepresent ahead of the vehicle.

If a marker object is not expected to be present, at decision step 408,the onboard data processing system can determine whether the detectedobject is identifiable. If the detected object is identifiable, forexample, if the processing system determines that the detected object isa pedestrian, the onboard data processing system can return to step 402to continue monitoring the surroundings. The onboard data processingsystem can also implement algorithms for situations involving encounterwith a pedestrian, such as by coming to a stop at a predetermineddistance from the pedestrian. If the detected object is notidentifiable, at step 412, the onboard data processing system cansuspend operation of the vehicle for safety reasons as described above.The vehicle can be stopped or slowed down to a stop when the operationof the vehicle is suspended.

If a marker object is expected to be present, at decision step 410, theonboard data processing system can determine whether the detected objectis the marker object. If the detected object is the marker object, theonboard data processing system can return to step 402 to continuemonitoring the surroundings. If the detected object is not the markerobject, the onboard data processing system can also proceed to step 412to suspend operation of the vehicle, as the onboard data processingsystem cannot ascertain whether the vehicle is still on the plannedroute.

At step 414, the onboard data processing system can send a request forintervention to the remote operator via a wireless transmitter. Thesequence of steps 412 and 414 can be reversed so that the processingsystem can send a request for intervention before stopping the vehicle.At step 416, the onboard data processing system can optionally verifythe identity of the remote server at which the remote operator islocated. If the identity of the remote server cannot be verified, atstep 418, the onboard data processing system can search for theappropriate remote server until the identity of a remote server isverified. For example, the onboard data processing system of the vehicleand/or the remote server can send encoded messages to each other fordecoding as a form of identity verification.

Once the identity of the remote server is verified, at step 420, theonboard data processing system can transmit the visual data detected bythe one or more sensors to the remote server. At step 422, the onboarddata processing system can receive a response from the remote server viaa wireless receiver. At step 424, the onboard data processing system canact upon the response. As will be described in greater detail below, theaction of the onboard data processing system can lead to an update ofthe operation of the vehicle. The update can include commanding thevehicle to remain stopped as the remote operator may determine that itis unsafe for the vehicle to move. The update can include commanding thevehicle to switch to the adjacent lane and continue on its planned routeif the remote operator determines that at least the adjacent lane issafe for traveling. The update can include commanding the vehicle tocontinue on its current lane if the remote operator determines that theunknown object is not a road hazard (for example, a paper bag or cardbox) and/or the vehicle is on its planned route (for example, the markertraffic light being located slightly outside the detection field of thesensor(s) on the vehicle).

FIG. 5 illustrates an example process 500 of receiving a remoteintervention request from the onboard data processing system of thevehicle disclosed herein and providing a response by the remote server.At step 502, the remote server can receive a request for remoteintervention by the onboard data processing system. At decision step504, the remote server can optionally verify the identity of thevehicle, such as using the methods described above. If the identity ofthe vehicle cannot be verified, the remote server can ignore the requestat step 506 as the remote server may not have authorization to provideremote intervention to the requesting vehicle. If the identity of thevehicle is verified, at step 508, the remote server can request thevisual data from the onboard data processing system. Upon receiving thevisual data, at decision step 510, the remote operator at the remoteserver can determine whether the object in the visual data isidentifiable. If the object cannot be immediately identified, at step512, the remote server can connect to additional knowledge bases, suchas the local traffic control data, traffic surveillance cameras, orothers, that may provide more information to help the remote operator inidentifying the object in the visual data or the marker object missingfrom the visual data. For example, by connecting to traffic surveillancecameras, the remote operator can locate the missing traffic light thatare outside the detection field of the one or more sensors of thevehicle. At step 514, the remote server can output the response to theonboard data processing system of the vehicle to provide remoteintervention.

Examples of Remote Operator Taking Control

In some embodiments, the remote operator can take over (for example,temporarily) control of operation of the vehicle from the onboard dataprocessing system of the vehicle. As shown in an example process 600 inFIG. 6A, upon the onboard data processing system requesting remoteintervention from a remote operator at a remote server at step 602(which can include any of the additional steps described above withreference to FIGS. 3 and 4), the onboard data processing system canreceive a direct command from the remote server at step 604. The directcommand can include a command that the onboard data processing systemhands over the control of operation of the vehicle to the remoteoperator, at least for a limited amount of time or for the task ofupdating the operation of the vehicle until the vehicle is no longerdisabled. At step 606, the onboard data processing system can relinquishthe control of operation of the vehicle to the remote operator at theremote server. After the remote operator has updated the operation ofthe vehicle, which can include any of the updates described above, theremote operator can turn the control of operation of the vehicle back tothe onboard data processing system and the onboard data processingsystem can regain the control at step 608.

As shown in the example process 650 in FIG. 6B, the remote server canreceive a request for remote intervention from the vehicle at step 652(which can include any of the additional steps described above withreference to FIGS. 3 and 5). The remote server can send a direct commandto the vehicle at step 654 to request for the control of operation ofthe vehicle. At step 656, the remote operator at the server can analyzethe visual data transmitted by the onboard data processing system of thevehicle. The remote operator can output a control signal to the vehicleat step 658. The control signal can command the vehicle to update itsoperation in any of the manners described above. At step 660, the remoteoperator can return the control of vehicle to the onboard dataprocessing system of the vehicle. In some embodiments, the remoteoperator can analyze the visual data before sending a direct command tothe vehicle. In some implementations, the remote operator can send adirect command to the vehicle to request the control of operation of thevehicle after having analyzed the visual data transmitted by thevehicle. The remote operator may take control of operation of thevehicle when the remote operator determines that it is safe for thevehicle to continue moving on its current lane and/or on an adjacentlane. The remote operator may send a “remain stopped” response to thevehicle without taking control of the vehicle when the remote operatordetermines that it is not safe for the vehicle to proceed.

Examples of Remote Operator Providing Decision-Making Assistance

In some embodiments, the remote operator may provide a decision-makingassistance response to the vehicle rather than taking over control ofthe vehicle when the vehicle requests for remote intervention. Theonboard data processing system of the vehicle can maintain full controlof operation of the vehicle, and can update vehicle operation based atleast in part on the decision-making assistance response from the remoteoperator. The onboard data processing system can also update the vehicleoperation based on aggregated information including the response fromthe remote operator and vehicle-based sensor data.

As shown in an example process 700 of FIG. 7A, the remote server canreceive a request for remote intervention at step 702 in the form of anobject identification request from any of the example vehicles describedabove. The remote server can request and receive visual data from thevehicle in any manner described above at step 704. The remote server cananalyze the visual data to identify the object in the visual data atstep 706. As described above, the remote server is capable of connectingto additional knowledge bases and/or resources that may help inidentifying the object in the visual data. At step 708, the remoteserver can output a response in the form of an identification of theobject.

As shown in an example process 750 of FIG. 7B, the remote server canreceive a request for remote intervention at step 752 in the form of adecision confirmation request from any of the example vehicles describedabove. The decision confirmation request can be, for example, a requestfor authorization to pass a solid line into an adjacent unblock lane, arequest to confirm that the unknown object is or is not a road hazard,or a request to confirm that the vehicle is on the planned route. Theremote server can request and receive visual data from the vehicle inany manner described above at step 754. The remote server can analyzethe visual data to identify the object in the visual data at step 756.As described above, the remote server is capable of connecting toadditional knowledge bases and/or resources that may help in analyzingthe visual data and confirming the decision suggested by the onboarddata processing system of the vehicle. At step 758, the remote servercan output a response in the form of a decision confirmation.

FIG. 8A illustrates an example onboard data processing system 800 of anyof the vehicle examples disclosed herein. As discussed above, theonboard data processing system 800 can receive a decision-makingassistance response 804 (such as an object identification and/or adecision confirmation) from a remote operator located at a remoteserver. The onboard data processing system 800 can also receive currentsensor data 802 from the one or more sensors disclosed herein. Theonboard data processing system 800 can output a control signal 806 toupdate the operation of the vehicle based on both the currentvehicle-based sensor data 802 and the decision-making assistanceresponse 804 from the remote operator. The onboard data processingsystem 800 can thus combine the more intelligent and more flexibledecision-making ability of the remote operator and the more currentand/or more real-time detection of the surroundings to improve theefficiency, accuracy, and safety of the final control signal in updatingthe operation of the vehicle. For example, after the remote operator hasdetermined that the indeterminate object is not a road hazard or it issafe to cross a solid line to the adjacent lane, the sensor(s) on thevehicle may detect additional road hazard or safety issues, such as apedestrian crossing the road in the meantime. The additional sensor datacan allow the onboard data processing system to adjust the controlsignal, such as to wait for the pedestrian to cross to the other side ofthe road before implementing the decision based on the response from theremote operator.

As shown in an example process 810 of FIG. 8B, the onboard dataprocessing system of the vehicle can update the operation of the vehicleusing remote intervention. The onboard data processing system canrequest for remote intervention from a remote server at step 812. Asdescribed above, the onboard data processing system can make such arequest when the operation of the vehicle is suspended or disabled, suchas in situations illustrated in FIGS. 2A-2C. The onboard data processingsystem can receive a response from the remote server at step 814. Theresponse can be a decision-making assistance response. At decision step816, the onboard data processing system can determine whether it is safeto proceed in the current lane or the adjacent lane. The onboard dataprocessing system can make this determination based on the response fromthe remote server. If the onboard data processing system determines thatit is not safe for the vehicle to proceed, at step 818, the onboard dataprocessing system can suspend operation of the vehicle so that thevehicle can slow down to a stop (for example, at a predetermineddistance away from the indeterminate object) or the vehicle can remainstopped. In some embodiments, the onboard data processing system of thevehicle can send the request for remote intervention after the vehiclehas come to substantially a stop. In some embodiments, the onboard dataprocessing system can send the request for intervention as the onboarddata processing system anticipates that the vehicle is about to bestopped (for example, as the vehicle is approaching the indeterminateobject or roadblock).

If the onboard data processing system determines that it is safe for thevehicle to proceed, at step 820, the onboard data processing system canreceive and process the signal inputs from the one or more sensors onthe vehicle. At decision step 822, the onboard data processing systemcan determine based on the vehicle-based current sensor data whether itis still safe to proceed on the current or adjacent lane. If the onboarddata processing system determines that it is not safe for the vehicle toproceed, the onboard data processing system can proceed to step 818 tosuspend operation of the vehicle so that the vehicle can slow down to astop or the vehicle can remain stopped. If the onboard data processingsystem determines that it is safe for the vehicle to proceed based onaggregated information from the remote operator and the sensors, theonboard data processing system can output a control signal to update theoperation of the vehicle at step 824. The control signal can move thevehicle in its current lane or move the vehicle to an adjacent lane.

Examples AI Operators

The remote operator can be a human operator or an AI operator. The AIoperator may be employed to more efficiently handle requests fromvehicles that are of the same or substantially similar nature as havingbeen handled by a human operator. The AI can automate the responseoutput based on the response from the human operator. The AI operatorcan be training, for example, via deep learning or a neural networkmodel. The human operator can be deployed to respond to new requests orto analyze new visual data.

In the example process 900 shown in FIG. 9, the AI operator can receiveand analyze visual data from the vehicle that requests remoteintervention at step 902. At decision 904, the AI operator can determinewhether similar visual data has been previously processed by a humanoperator. For example, the AI operator can compare the current visualdata with previously processed visual data by comparing the visual datapixel by pixel, and/or by comparing the time and/or location (such ascross roads) of the requesting vehicle. If the visual data is notsimilar to any previously processed visual data (such as when the objectin the visual data is different than the object in the previouslyprocessed visual data), the AI operator can forward the request to ahuman operator at step 906. If the visual data is similar or the same toany previously processed visual data, such as when both sets of visualdata show the same object, the AI operator can retrieve the responsepreviously provided by the human operator at step 908. The AI operatorcan output a response based on the previously sent response and thenature of the request at step 910. The response outputted by the AIoperator can be of the same or different nature as the previousresponse. In some embodiments, the response outputted by the AI operatorcan depend on the nature of the request from the vehicle and can be adirect command or a decision-making assistance response. The use of AIoperators for processing similar requests or similar visual data setscan improve the efficiency in providing remote intervention to thevehicles, and thereby reducing the standby time when the operation ofthe vehicles are suspended or disabled.

In regard to the figures described herein, other embodiments arepossible within the scope of the present invention, such that theabove-recited components, steps, blocks, operations, and/ormessages/requests/queries/instructions are differently arranged,sequenced, sub-divided, organized, and/or combined. In some embodiments,a different component may initiate or execute a given operation. Forexample, in some embodiments, the control signal to update the operationof the vehicle can be directed to a controller of the wheels and/or acontroller of the brake of the vehicle, which may be separatecontrollers from the onboard data processing system, so as to move thevehicle or cause the vehicle to remain stopped.

Example Embodiments

Some example enumerated embodiments of the present invention are recitedin this section in the form of methods, systems, and non-transitorycomputer-readable media, without limitation.

In some embodiments, a vehicle configured to receive remote interventioncan comprise an onboard hardware processor, at least one sensor inelectronic communication with the onboard hardware processor, the atleast one sensor configured to detect visual data about a road on whichthe vehicle is traveling and transmitting the visual data to the onboardhardware processor, a wireless transmitter, and a wireless receiver,wherein the onboard hardware processor can be configured to transmit aremote intervention request to a remote server via the wirelesstransmitter in response to an operation of the vehicle being suspended,transmit the visual data to the remote server, receive a response fromthe remote server in response to the remote intervention request via thewireless receiver, and act upon the response, the acting resulting in acontrol signal to update the operation of the vehicle.

In some embodiments, the vehicle can incorporate one or more of thefollowing features: the visual data can comprise three-dimensional imagedata; the operation of the vehicle can be suspended upon the visual datacontaining an indeterminate object; the indeterminate object cancomprise an unidentifiable object; the indeterminate object can comprisea missing marker object; the operation of the vehicle can be suspendedupon a currently lane traveled by the vehicle being blocked and thecurrent lane being separated from an adjacent lane by a solid line; theonboard hardware processor can stop or slow down the vehicle in responseto the operation of the vehicle being suspended; the vehicle can be anautonomous driving vehicle; the response can comprise a direct commandto the onboard hardware processor; the onboard hardware processor can beconfigured to act upon the direct command by relinquishing control ofthe vehicle to the remote server, wherein the remote server can beconfigured to output the control signal to the vehicle; the response cancomprise a classification of the indeterminate object or a confirmationof a decision suggested by the onboard hardware processor; the onboardhardware processor can be configured to output a control signal toupdate the operation of the vehicle based at least in part on theresponse; the onboard hardware processor can output the control signalbased at least in part on additional signal inputs from the at least onesensor; the at least one sensor can comprise LiDAR, camera, infraredcamera, ultrasonic transducer, Radar, or a combination thereof; theresponse from the remote server can be provided by a human operatorviewing the visual data; the response from the remote server can beprovided by an artificial intelligence (AI) operator viewing the visualdata; the response provided by the AI operator can be based on a priorresponse provided by a human operator; the vehicle can move to anadjacent free lane in response to the control signal; the vehicle canmove in its current lane in response to the control signal; the vehiclescan remain stopped in response to the control signal.

In some embodiments, a method of providing remote invention by a remoteserver to a vehicle can comprise using an onboard hardware processor ofthe vehicle, receiving signal inputs from at least one sensor of thevehicle in electronic communication with the onboard hardware processor,the at least one sensor configured to detect visual data about a road onwhich the vehicle is traveling; outputting a remote intervention requestto a remote server via a wireless transmitter in response to anoperation of the vehicle being suspended; transmitting the visual datato the remote server; receiving a response from the remote server inresponse to the remote intervention request via a wireless receiver; andacting upon the response, the acting resulting in a control signal toupdate the operation of the vehicle.

In some embodiments, the method can incorporate one or more of thefollowing features: the visual data can comprise three-dimensional imagedata; the operation of the vehicle can be suspended upon the visual datacontaining an indeterminate object; the indeterminate object cancomprise an unidentifiable object; the indeterminate object can comprisea missing marker object; the operation of the vehicle can be suspendedupon a currently lane traveled by the vehicle being blocked and thecurrent lane being separated from an adjacent lane by a solid line; themethod can further comprise stopping or slowing down the vehicle inresponse to the operation of the vehicle being suspended; the vehiclecan be an autonomous driving vehicle; the response can comprise a directcommand to the onboard hardware processor; the acting can compriserelinquishing control of the vehicle to the remote server, wherein theremote server can be configured to output the control signal to thevehicle; the response can comprise a classification of the indeterminateobject or a confirmation of a decision suggested by the onboard hardwareprocessor; the acting can comprise outputting a control signal to updatethe operation of the vehicle based at least in part on the response; themethod can further comprise outputting the control signal based at leastin part on additional signal inputs from the at least one sensor; the atleast one sensor can comprise LiDAR, camera, infrared camera, ultrasonictransducer, Radar, or a combination thereof; the response from theremote server can be provided by a human operator viewing the visualdata; the response from the remote server can be provided by anartificial intelligence (AI) operator viewing the visual data; theresponse provided by the AI operator can be based on a prior responseprovided by a human operator; the vehicle can move to an adjacent freelane in response to the control signal; the vehicle can move in itscurrent lane in response to the control signal; the vehicles can remainstopped in response to the control signal.

In some embodiments, a configured to update vehicle operation inresponse to remote intervention can comprise an onboard hardwareprocessor, at least one sensor in electronic communication with theonboard hardware processor, the at least one sensor configured to detectvisual data about a road on which the vehicle is traveling andtransmitting, on a regular basis, the visual data to the onboardhardware processor, a wireless transmitter, and a wireless receiver,wherein the onboard hardware processor can be configured to transmit arequest for decision-making assistance to a remote server via thewireless transmitter in response to an operation of the vehicle beingsuspended, transmit the visual data to the remote server, receive adecision-making assistance response from the remote server in responseto the request via the wireless receiver, and output a control signal toupdate the operation of the vehicle based on an aggregated informationcomprising the response and the visual data transmitted on the regularbasis from the at least one sensor.

In some embodiments, the vehicle can incorporate one or more of thefollowing features: the request for decision-making assistance cancomprise an object classification request or a decision confirmationrequest; the decision-making assistance response can comprise an objectclassification response or a decision confirmation response; the visualdata can comprise three-dimensional image data; the operation of thevehicle can be suspended upon the visual data containing anindeterminate object; the object classification request can comprise arequest to identify an unknown object; the decision confirmation requestcan comprise a request to authorize moving to an adjacent lane separatedfrom a current lane by a solid line; the decision confirmation requestcan comprise a request to confirm whether it is safe to continuetraveling in current lane; the decision confirmation request cancomprise a request to confirm presence of a missing marker object; theonboard hardware processor can stop or slow down the vehicle in responseto the operation of the vehicle being suspended; the vehicle can be anautonomous driving vehicle; the response can comprise a direct commandto the onboard hardware processor; the onboard hardware processor can beconfigured to act upon the direct command by relinquishing control ofthe vehicle to the remote server, wherein the remote server can beconfigured to output the control signal to the vehicle; the response cancomprise a classification of the indeterminate object or a confirmationof a decision suggested by the onboard hardware processor; the onboardhardware processor can be configured to output a control signal toupdate the operation of the vehicle based at least in part on theresponse; the onboard hardware processor can output the control signalbased at least in part on additional signal inputs from the at least onesensor; the at least one sensor can comprise LiDAR, camera, infraredcamera, ultrasonic transducer, Radar, or a combination thereof; theresponse from the remote server can be provided by a human operatorviewing the visual data; the response from the remote server can beprovided by an artificial intelligence (AI) operator viewing the visualdata; the response provided by the AI operator can be based on a priorresponse provided by a human operator; the vehicle can move to anadjacent free lane in response to the control signal; the vehicle canmove in its current lane in response to the control signal; the vehiclescan remain stopped in response to the control signal; the at least onesensor can continuously transmit the visual data to the onboard hardwareprocessor.

In some embodiments, a method of updating vehicle operation in responseto remote intervention can comprise using an onboard hardware processorof the vehicle, receiving, on a regular basis, signal inputs from atleast one sensor of the vehicle in electronic communication with theonboard hardware processor, the at least one sensor configured to detectvisual data about a road on which the vehicle is traveling; outputting arequest for decision-making assistance to a remote server via a wirelesstransmitter in response to an operation of the vehicle being suspended;transmitting the visual data to the remote server; receiving adecision-making assistance response from the remote server in responseto the request via a wireless receiver; and outputting a control signalto update the operation of the vehicle based on an aggregatedinformation comprising the response and the visual data transmitted onthe regular basis from the at least one sensor.

In some embodiments, the method can incorporate one or more of thefollowing features: the request for decision-making assistance cancomprise an object classification request or a decision confirmationrequest; the decision-making assistance response can comprise an objectclassification response or a decision confirmation response; the visualdata can comprise three-dimensional image data; the operation of thevehicle can be suspended upon the visual data containing anindeterminate object; the object classification request can comprise arequest to identify an unknown object; the decision confirmation requestcan comprise a request to authorize moving to an adjacent lane separatedfrom a current lane by a solid line; the decision confirmation requestcan comprise a request to confirm whether it is safe to continuetraveling in current lane; the decision confirmation request cancomprise a request to confirm presence of a missing marker object; themethod can further comprise stopping or slowing down the vehicle inresponse to the operation of the vehicle being suspended; the vehiclecan be an autonomous driving vehicle; the response can comprise a directcommand to the onboard hardware processor; the onboard hardwareprocessor can be configured to act upon the direct command byrelinquishing control of the vehicle to the remote server, wherein theremote server can be configured to output the control signal to thevehicle; the response can comprise a classification of the indeterminateobject or a confirmation of a decision suggested by the onboard hardwareprocessor; the onboard hardware processor can be configured to output acontrol signal to update the operation of the vehicle based at least inpart on the response; the onboard hardware processor can output thecontrol signal based at least in part on additional signal inputs fromthe at least one sensor; the at least one sensor can comprise LiDAR,camera, infrared camera, ultrasonic transducer, Radar, or a combinationthereof; the response from the remote server can be provided by a humanoperator viewing the visual data; the response from the remote servercan be provided by an artificial intelligence (AI) operator viewing thevisual data; the response provided by the AI operator can be based on aprior response provided by a human operator; the vehicle can move to anadjacent free lane in response to the control signal; the vehicle canmove in its current lane in response to the control signal; the vehiclescan remain stopped in response to the control signal; the receiving cancomprise continuously receiving the visual data from the at least onesensor.

In other embodiments, a system or systems may operate according to oneor more of the methods and/or computer-readable media recited in thepreceding paragraphs. In yet other embodiments, a method or methods mayoperate according to one or more of the systems and/or computer-readablemedia recited in the preceding paragraphs. In yet more embodiments, acomputer-readable medium or media, excluding transitory propagatingsignals, may cause one or more computing devices having one or moreprocessors and non-transitory computer-readable memory to operateaccording to one or more of the systems and/or methods recited in thepreceding paragraphs.

Terminology

Conditional language, such as, among others, “can,” “could,” “might,” or“may,” unless specifically stated otherwise, or otherwise understoodwithin the context as used, is generally intended to convey that certainembodiments include, while other embodiments do not include, certainfeatures, elements and/or steps. Thus, such conditional language is notgenerally intended to imply that features, elements and/or steps are inany way required for one or more embodiments or that one or moreembodiments necessarily include logic for deciding, with or without userinput or prompting, whether these features, elements and/or steps areincluded or are to be performed in any particular embodiment.

Unless the context clearly requires otherwise, throughout thedescription and the claims, the words “comprise,” “comprising,” and thelike are to be construed in an inclusive sense, as opposed to anexclusive or exhaustive sense, i.e., in the sense of “including, but notlimited to.” As used herein, the terms “connected,” “coupled,” or anyvariant thereof means any connection or coupling, either direct orindirect, between two or more elements; the coupling or connectionbetween the elements can be physical, logical, or a combination thereof.Additionally, the words “herein,” “above,” “below,” and words of similarimport, when used in this application, refer to this application as awhole and not to any particular portions of this application. Where thecontext permits, words using the singular or plural number may alsoinclude the plural or singular number respectively. The word “or” inreference to a list of two or more items, covers all of the followinginterpretations of the word: any one of the items in the list, all ofthe items in the list, and any combination of the items in the list.Likewise the term “and/or” in reference to a list of two or more items,covers all of the following interpretations of the word: any one of theitems in the list, all of the items in the list, and any combination ofthe items in the list.

In some embodiments, certain operations, acts, events, or functions ofany of the algorithms described herein can be performed in a differentsequence, can be added, merged, or left out altogether (e.g., not allare necessary for the practice of the algorithms). In certainembodiments, operations, acts, functions, or events can be performedconcurrently, e.g., through multi-threaded processing, interruptprocessing, or multiple processors or processor cores or on otherparallel architectures, rather than sequentially.

Systems and modules described herein may comprise software, firmware,hardware, or any combination(s) of software, firmware, or hardwaresuitable for the purposes described. Software and other modules mayreside and execute on servers, workstations, personal computers,computerized tablets, PDAs, and other computing devices suitable for thepurposes described herein. Software and other modules may be accessiblevia local computer memory, via a network, via a browser, or via othermeans suitable for the purposes described herein. Data structuresdescribed herein may comprise computer files, variables, programmingarrays, programming structures, or any electronic information storageschemes or methods, or any combinations thereof, suitable for thepurposes described herein. User interface elements described herein maycomprise elements from graphical user interfaces, interactive voiceresponse, command line interfaces, and other suitable interfaces.

Further, processing of the various components of the illustrated systemscan be distributed across multiple machines, networks, and othercomputing resources. Two or more components of a system can be combinedinto fewer components. Various components of the illustrated systems canbe implemented in one or more virtual machines, rather than in dedicatedcomputer hardware systems and/or computing devices. Likewise, the datarepositories shown can represent physical and/or logical data storage,including, e.g., storage area networks or other distributed storagesystems. Moreover, in some embodiments the connections between thecomponents shown represent possible paths of data flow, rather thanactual connections between hardware. While some examples of possibleconnections are shown, any of the subset of the components shown cancommunicate with any other subset of components in variousimplementations.

Embodiments are also described above with reference to flow chartillustrations and/or block diagrams of methods, apparatus (systems) andcomputer program products. Each block of the flow chart illustrationsand/or block diagrams, and combinations of blocks in the flow chartillustrations and/or block diagrams, may be implemented by computerprogram instructions. Such instructions may be provided to a processorof a general purpose computer, special purpose computer,specially-equipped computer (e.g., comprising a high-performancedatabase server, a graphics subsystem, etc.) or other programmable dataprocessing apparatus to produce a machine, such that the instructions,which execute via the processor(s) of the computer or other programmabledata processing apparatus, create means for implementing the actsspecified in the flow chart and/or block diagram block or blocks. Thesecomputer program instructions may also be stored in a non-transitorycomputer-readable memory that can direct a computer or otherprogrammable data processing apparatus to operate in a particularmanner, such that the instructions stored in the computer-readablememory produce an article of manufacture including instruction meanswhich implement the acts specified in the flow chart and/or blockdiagram block or blocks. The computer program instructions may also beloaded to a computing device or other programmable data processingapparatus to cause operations to be performed on the computing device orother programmable apparatus to produce a computer implemented processsuch that the instructions which execute on the computing device orother programmable apparatus provide steps for implementing the actsspecified in the flow chart and/or block diagram block or blocks.

Any patents and applications and other references noted above, includingany that may be listed in accompanying filing papers, are incorporatedherein by reference. Aspects of the invention can be modified, ifnecessary, to employ the systems, functions, and concepts of the variousreferences described above to provide yet further implementations of theinvention. These and other changes can be made to the invention in lightof the above Detailed Description. While the above description describescertain examples of the invention, and describes the best modecontemplated, no matter how detailed the above appears in text, theinvention can be practiced in many ways. Details of the system may varyconsiderably in its specific implementation, while still beingencompassed by the invention disclosed herein. As noted above,particular terminology used when describing certain features or aspectsof the invention should not be taken to imply that the terminology isbeing redefined herein to be restricted to any specific characteristics,features, or aspects of the invention with which that terminology isassociated. In general, the terms used in the following claims shouldnot be construed to limit the invention to the specific examplesdisclosed in the specification, unless the above Detailed Descriptionsection explicitly defines such terms. Accordingly, the actual scope ofthe invention encompasses not only the disclosed examples, but also allequivalent ways of practicing or implementing the invention under theclaims.

To reduce the number of claims, certain aspects of the invention arepresented below in certain claim forms, but the applicant contemplatesother aspects of the invention in any number of claim forms. Forexample, while only one aspect of the invention is recited as ameans-plus-function claim under 35 U.S.C sec. 112(f) (AIA), otheraspects may likewise be embodied as a means-plus-function claim, or inother forms, such as being embodied in a computer-readable medium. Anyclaims intended to be treated under 35 U.S.C. § 112(f) will begin withthe words “means for,” but use of the term “for” in any other context isnot intended to invoke treatment under 35 U.S.C. § 112(f). Accordingly,the applicant reserves the right to pursue additional claims afterfiling this application, in either this application or in a continuingapplication.

What is claimed is:
 1. A vehicle configured to update vehicle operationin response to remote intervention, the vehicle comprising: an onboardhardware processor; at least one sensor in electronic communication withthe onboard hardware processor, the at least one sensor configured todetect visual data about a road on which the vehicle is traveling andtransmitting, on a regular basis, the visual data to the onboardhardware processor; a wireless transmitter; and a wireless receiver,wherein the onboard hardware processor is configured to: transmit arequest for decision-making assistance to a remote server via thewireless transmitter in response to an operation of the vehicle beingsuspended, transmit the visual data to the remote server, receive adecision-making assistance response from the remote server in responseto the request via the wireless receiver, and output a control signal toupdate the operation of the vehicle based on an aggregated informationcomprising the response and the visual data transmitted on the regularbasis from the at least one sensor.
 2. The vehicle of claim 1, whereinthe decision-making assistance response comprises an objectclassification response or a decision confirmation response.
 3. Thevehicle of claim 1, wherein the visual data comprises three-dimensionalimage data.
 4. The vehicle of claim 1, wherein the object classificationrequest comprises a request to identify an unknown object.
 5. Thevehicle of claim 1, wherein the decision confirmation request comprisesa request to authorize moving to an adjacent lane separated from acurrent lane by a solid line.
 6. The vehicle of claim 1, wherein thedecision confirmation request comprises a request to confirm whether itis safe to continue traveling in current lane.
 7. The vehicle of claim1, wherein the decision confirmation request comprises a request toconfirm presence of a missing marker object.
 8. The vehicle of claim 1,wherein the vehicle is an autonomous driving vehicle.
 9. The vehicle ofclaim 1, wherein the at least one sensor comprises LiDAR, camera,infrared camera, ultrasonic transducer, Radar, or a combination thereof.10. The vehicle of claim 1, wherein the response from the remote serveris provided by an artificial intelligence (AI) operator based on a priorresponse provided by a human operator.
 11. The vehicle of claim 1,wherein the vehicle moves to an adjacent free lane or in its currentlane in response to the control signal.
 12. The vehicle of claim 1,wherein the vehicles remains stopped in response to the control signal.13. A method of updating vehicle operation in response to remoteintervention, the method comprising: using an onboard hardware processorof the vehicle: receiving, on a regular basis, signal inputs from atleast one sensor of the vehicle in electronic communication with theonboard hardware processor, the at least one sensor configured to detectvisual data about a road on which the vehicle is traveling; outputting arequest for decision-making assistance to a remote server via a wirelesstransmitter in response to an operation of the vehicle being suspended;transmitting the visual data to the remote server; receiving adecision-making assistance response from the remote server in responseto the request via a wireless receiver; and outputting a control signalto update the operation of the vehicle based on an aggregatedinformation comprising the response and the visual data transmitted onthe regular basis from the at least one sensor.
 14. The method of claim13, wherein the decision-making assistance response comprising an objectclassification response or a decision confirmation response.
 15. Themethod of claim 13, wherein the visual data comprises three-dimensionalimage data.
 16. The method of claim 13, wherein the objectclassification request comprises a request to identify an unknownobject.
 17. The method of claim 13, wherein the decision confirmationrequest comprises a request to authorize moving to an adjacent laneseparated from a current lane by a solid line.
 18. The method of claim13, wherein the decision confirmation request comprises a request toconfirm whether it is safe to continue traveling in current lane. 19.The method of claim 13, wherein the decision confirmation requestcomprises a request to confirm the presence of a missing marker object.20. The method of claim 13, wherein the vehicle is an autonomous drivingvehicle.
 21. The method of claim 13, wherein the at least one sensorcomprises LiDAR, camera, infrared camera, ultrasonic transducer, Radar,or a combination thereof.
 22. The method of claim 13, wherein theresponse from the remote server is provided by an artificialintelligence (AI) operator based on a prior response provided by a humanoperator.
 23. The method of claim 13, wherein the vehicle moves to anadjacent free lane or in its current lane in response to the controlsignal.
 24. The method of claim 13, wherein the vehicles remains stoppedin response to the control signal.